Mining Network Logs: Information Quality Challenges

Network logs are the key to many critical functions such as network security, network monitoring and network management. They play an important role in intrusion detection and early warning for potential worm and virus attacks. However, network log data are under-utilized -largely ignored until the occurrence of an event that requires back tracking for diagnostic purposes. There are two main reasons why network logs are not subject to more rigorous analysis – the sheer volume and the inherent information quality challenges. In this paper, we use the context of classical data quality principles to outline some of the issues that we encountered, and the solutions that we devised, while working on a real network management application involving large amounts of network log data. While our discussion is centered on our case study, the problems we encounter and the solutions we devise are general and apply to a wide array of network log data and applications.